The present invention relates to decoding control for enhancing the freedom of a process to edit a moving picture.
An MPEG (Moving Picture Expert Group) technique is known as a typical method for compressing a moving picture, that is, for reducing the amount of data representing the moving picture, by carrying out a predictive encoding process on any specific one of frames, which compose the moving picture, with reference to frames other than the specific one.
In accordance with the MPEG technique, data is subjected to quantization processing in an encoding process and encoded data is subjected to inverse-quantization processing in a decoding process. Thus, if the encoding and decoding processes are carried out repeatedly, the quality of a picture obtained as a result of the last decoding process is poor.
In order to solve this problem, in an operation to edit a moving picture, which has been obtained as a result of an MPEG encoding process, the so-called edit points are set on an MPEG stream (or encoded data), which serves as an editing raw material representing the moving picture, without decoding the MPEG stream. The edit points are IN points and OUT points. Then, a list of edit points set on the MPEG stream is made. The list is referred to hereafter as an edit-point list. In an operation to reproduce a result of the editing operation, the MPEG stream is decoded in accordance with edit points shown on the edit-point list.
The aforementioned MPEG stream comprises GOPs each including a plurality of pictures. Symbol GOP (m) denotes an mth GOP as counted from the beginning of the MPEG stream. Assume for example that a start sequence of GOPs beginning with GOPi not shown in the figure and ending with GOP (m) on an MPEG stream 201 shown in FIG. 1 has been reproduced. Then, GOP (n) located after GOP (m) is reproduced. Finally, another sequence of GOPs beginning with GOP (m+2) located before GOP (n) and ending with GOPj not shown in the figure is reproduced. The operations to reproduce the start sequence, GOP (n) and the other sequence is referred to as reproduction processing based on the so-called cut editing process. In this cut editing process, an edit-point list 202 shown in FIG. 1 is made. It is to be noted that the edit-point list 202 is shown only in a conceptual manner focusing on edit points. Details of the edit-point list 202 will be explained later.
In the edit-point list 202 shown in FIG. 1, an edit point Ti represents the start position of GOPi, which is reproduced as the first GOP. The edit point T1 represents the end position of GOP (m) or the start position of GOP (m+1). By the same token, an edit point T2 represents the end position of GOP (m+1) or the start position of GOP (m+2). In the same way, an edit point T3 represents the end position of GOP (n−1) or the start position of GOP (n). Similarly, an edit point T4 represents the end position of GOP (n) or the start position of GOP. (n+1). In addition, an edit point Tj represents the start position of GOPj, which is reproduced as the last GOP. It is to be noted that the edit points T1 and T4 are each an OUT point while the edit points T2 and T3 are each an IN point.
Notation ‘Ti→T1’ on the first line of the edit-point list 202 shown in FIG. 1 indicates that the MPEG stream 201 is to be reproduced from the position (time code) Ti on the MPEG stream 201 to the time code T1. By the same token, notation ‘T3→T4’ on the second line indicates that the MPEG stream 201 is to be reproduced the time code T3 to the time code T4. In the same way, notation ‘T2→Tj’ on the third line indicates that the MPEG stream 201 is to be reproduced the time code T2 to the time code Tj.
Thus, in accordance with the edit-point list 202 shown in FIG. 1, the start sequence of GOPs beginning with GOPi and ending with GOP (m) on an MPEG stream 201 shown in FIG. 1 is first reproduced. Then, GOP (n) located after GOP (m+2) and GOPj is reproduced. Finally, the other sequence of GOPs beginning with GOP (m+2) and ending with GOPj is reproduced.
As described above, the edit-point list 202 is first made in an editing operation and the MPEG stream 201 is then reproduced in a reproduction operation. Thus, by carrying out an editing operation without decoding the MPEG stream 201, it is possible to implement a desired editing operation and, hence, obtain a desired result of editing.
FIG. 1 is a diagram showing an operation to edit only one MPEG stream as a raw editing material. It is to be noted, however, that a plurality of MPEG streams can also be used as a raw editing material to be edited in the same way as that shown in FIG. 1.
By the way, in an operation to reproduce an MPEG stream, which is obtained as a result to edit a raw MPEG stream by making an edit-point list as described above, it is necessary to decode the resulting MPEG stream, that is, to put the resulting MPEG stream in an MPEG decoding process. Assume that an MPEG decoder for decoding the MPEG stream has a decoding-process speed by one. In this case, by using only a unit of MPEG decoder having such a speed, it will be difficult to seamlessly reproduce an MPEG stream, which is obtained as a result of a process to edit a raw MPEG stream by making an edit-point list as described above.
In accordance with the MPEG technique, there are provided three picture types, namely, an Intra (I) picture, a Predictive (P) picture and a Bidirectionally predictive (B) picture. An I picture is a picture to be subjected to an intra encoding process without referencing other pictures. A P picture is a picture to be subjected to an intra encoding process or a predictive encoding process, which is an encoding process carried out by referencing an I picture or another P picture and using a predicted picture generated from the referenced picture. The referenced picture is a picture displayed at a time preceding the P picture being processed. A B picture is a picture to be subjected to an intra encoding process or a bidirectionally predictive encoding process, which is an encoding process carried out by referencing an I picture or another P picture and using a predicted picture generated from the referenced picture. The referenced picture in the bidirectionally predictive encoding process is a picture displayed at a time preceding or succeeding the P picture being processed. It is to be noted that, while an I or P picture can be used as a picture to be referenced in a predictive encoding process or a bidirectionally predictive encoding process, a B picture is never used as a picture to be referenced in any encoding process.
As an example, consider a long GOP with a structure consisting of 15 pictures (or 15 frames) wherein each of I and P pictures is placed as the last one of three consecutive pictures including two other B pictures.
Let alphabetical characters I, P and B denote respectively the I, P and B picture types of the pictures composing a GOP. In this case, an example of a GOP is represented by a list as follows:
B1, B2, I3, B4, B5, P6, B7, B8, P9, B10, B11, P12, B13, B14, P15
where a numerical subscript appended to each of the alphabetical characters I, P and B denotes an order number of an order along the time axis.
As described above, a B picture is a picture encoded by referencing not only a picture displayed at a time preceding the B picture, but also a picture displayed at a time succeeding the B picture. Thus, a B picture cannot be decoded unless the picture used as a referenced picture to be displayed at a time succeeding the B picture is decoded before the B picture in a decoding operation.
In accordance with the MPEG technique, a picture used as a referenced picture in an operation to encode a B picture is encoded before the B picture. By the same token, a picture used as a referenced picture in an operation to decode B picture is decoded before the B picture. In the case of the aforementioned list representing the GOP consisting of pictures ranging from B1 to P15, the pictures are encoded and decoded in the following order:
I3, B1, B2, P6, B4, B5, P9, B7, B8, P12, B10, B11, P15, B13, B14.
To put it in detail, pictures B1 and B2 are each encoded and decoded by referencing picture P15 displayed at a preceding time and picture I3 displayed at a succeeding time. Picture P15 is a picture pertaining to the immediately preceding GOP while picture I3 is a picture pertaining to the same GOP.
Picture I3 is encoded and decoded without referencing another picture.
Pictures B4 and B5 are each encoded and decoded by referencing picture I3 displayed at a preceding time and picture P6 displayed at a succeeding time. Pictures I3 and P6 are each a picture pertaining to the same GOP.
Picture P6 is encoded and decoded by referencing picture I3 displayed at a preceding time. Picture I3 is a picture pertaining to the same GOP.
Pictures B7 and B8 are each encoded and decoded by referencing picture P6 displayed at a preceding time and picture P9 displayed at a succeeding time. Pictures P6 and P9 are each a picture pertaining to the same GOP.
Picture P9 is encoded and decoded by referencing picture P6 displayed at a preceding time. Picture P6 is a picture pertaining to the same GOP.
Pictures B10 and B11 are each encoded and decoded by referencing picture P9 displayed at a preceding time and picture P12 displayed at a succeeding time. Pictures P9 and P12 are each a picture pertaining to the same GOP.
Picture P12 is encoded and decoded by referencing picture P9 displayed at a preceding time. Picture P9 is a picture pertaining to the same GOP.
Pictures B13 and B14 are each encoded and decoded by referencing picture P12 displayed at a preceding time and picture P15 displayed at a succeeding time. Pictures P12 and P15 are each a picture pertaining to the same GOP.
Picture P15 is encoded and decoded by referencing picture P12 displayed at a preceding time. Picture P12 is a picture pertaining to the same GOP.
As described above, pictures B1 and B2 of any GOP (n) are encoded and decoded by referencing an I or P picture displayed at a succeeding time and an I or P picture displayed at a preceding time. In this case, the picture displayed at the succeeding time is picture I3 pertaining to the same GOP (n) but the picture displayed at the preceding time is picture P15 pertaining to GOP (n−1) immediately preceding GOP (n).
Assume that the current edit point is set at the head of GOP (n). In this case, in order to carry out a reproduction process starting from the head of GOP (n), GOP (n) cannot be decoded unless GOP (n−1) immediately preceding GOP (n) is decoded first as is obvious from a picture-decoding order 203 shown in FIG. 2.
To put it concretely, in order to decode pictures B1 and B2 pertaining to GOP (n), it is necessary to reference picture P15 of GOP (n−1) immediately preceding GOP (n). In accordance with the conventional technology, an MPEG decoder is controlled in GOP units so that GOP (n) cannot be decoded unless GOP (n−1) immediately preceding GOP (n) is decoded first.
In accordance with the conventional technology, only the pictures composing GOP (n) are displayed without displaying the pictures composing GOP (n−1) as is obvious from a picture-decoding order 204 shown in FIG. 2 in spite of the fact that both GOP (n) and GOP (n−1) are decoded.
As described above, the MPEG decoder needs to decode GOP (n−1), which will not be displayed, in addition to GOP (n). Thus, in order to obtain editing results obtained earlier by referring to FIG. 1, GOP (n) is decoded after GOP (m) separated away from GOP (n) is decoded but, in order to decode GOP (n), it is necessary to decode GOP (n−1) immediately preceding GOP (n). Accordingly, by using only a unit of MPEG decoder having a 1-time decoding-process speed, there is no time margin to decode GOP (n−1) after decoding GOP (m) but before decoding GOP (n).
As a result, by using only a unit of MPEG decoder having a 1-time decoding-process speed, on the basis of an edit-point list, it is impossible to reproduce results of editing an MPEG screen.
Japanese Patent Laid-open No. 2002-101379 discloses a method to reproduce results of editing an MPEG screen by using two MPEG decoders on the basis of an edit-point list.
In accordance with the method disclosed in Japanese Patent Laid-open No. 2002-101379, a duration required between a start edit point and an end edit point must be at least the length of a GOP. By using two MPEG decoders to alternately decode an MPEG stream, which has been edited under such a condition, it is possible to reproduce results of editing the MPEG screen. The start and end edit points can each be an IN or OUT point. If the start edit point is an IN point, the end edit point is an OUT point. If the start edit point is an OUT point, on the other hand, the end edit point is an IN point. In the following description, the minimum duration is appropriately referred to as a shortest cut duration or a margin.
An MPEG stream 211 shown in FIG. 3 is the same as the MPEG stream 201 shown in FIG. 1. Assume that a start sequence of GOPs beginning with GOPi not shown in the figure and ending with GOP (m) on an MPEG stream 211 is to be reproduced. Then, GOP (n) located after GOP (m) is to be reproduced. Finally, another sequence of GOPs beginning with GOP (m+2) located before GOP (n) and ending with GOPj not shown in the figure is to be reproduced. In order to reproduce these GOPs, a cut editing process is carried out to set edit points T1, T2, T3 and T4.
In this case, in an operation to reproduce these results of the editing, the start sequence of GOPs ending with picture P15 at the tail of GOP (m) is first displayed to be followed by pictures B1 to P15 of GOP (n) before displaying the other sequence of GOPs beginning with picture B1 at the head of GOP (m+2) as is obvious from a picture-displaying order 212 shown in FIG. 3. At the tail of GOP (m), the edit point T1 has been set as an OUT point. At pictures B1 and P15 of GOP (n), the edit points T3 and T4 have been set as IN and OUT points respectively. At the head of GOP (m+2), the edit point T2 has been set as an IN point.
In order to obtain a picture-displaying order 212 shown in FIG. 3, in accordance with the method disclosed in Japanese Patent Laid-open No. 2002-101379, a specific one of 2 MPEG decoders #1 and #2, is used to decode all pictures ending with GOP (m) as shown in a picture-decoding order 213 of FIG. 3. For example, the specific MPEG decoder is MPEG decoder #1. It is to be noted that, in the process to decode GOP (m), the processing to decode picture B14 to be decoded last among the pictures composing GOP (m) must be completed by a time t2, at which picture B14 is to be displayed. Thus, the process to decode GOP (m) must be started before a time leading ahead of the time t2 by a period of time it takes to display pictures composing a GOP. In the picture-decoding order 213 shown in FIG. 3, a time t1 is determined to be the time to start the process to decode GOP (m).
The edit point T1 is reached after decoder #1 has decoded GOP (m) to be displayed. The process then is continued to processing to decode and display GOP (n). The processing to decode and display GOP (n) must start at the edit point T3 and end at the edit point T4.
In order to decode GOP (n), it is necessary to decode GOP (n−1) immediately preceding GOP (n) as described earlier. In the process to decode GOP (n), the processing to decode picture B14 to be decoded last among the pictures composing GOP (n) must be completed by a time t3, at which picture B14 is to be displayed. Thus, the process to decode GOP (n) must be started before a time leading ahead of the time t3 by a period of time it takes to display pictures composing a GOP. In a picture-decoding order 214 shown in FIG. 3, a time t2 is determined to be the time to start the process to decode GOP (n). By the same token, the process to decode GOP (n−1) immediately preceding GOP (n) must be started before a time leading ahead of the time t2 by a period of time it takes to display pictures composing a GOP in the picture-decoding order 214 shown in FIG. 3, a time t1 is determined to be the time to start the process to decode GOP (n−1).
For the reason described above, MPEG decoder #2 starts decoding GOP (n−1) at the time t1 and continues the process by decoding GOP (n) as shown in the picture-decoding order 214 of FIG. 3. MPEG decoder #2 is the other one of two decoders #1 and #2.
In the mean time, MPEG decoder #1 should naturally terminate the decoding process at the end of the processing to decode GOP (m). After GOP (n) is decoded by decoder #2 and displayed, however, the edit point T4 is reached. Then, MPEG decoder #1 must decode the other sequence of GOPs beginning with GOP (m+2) at the edit point T2 for a display.
In order to decode GOP (m+2), it is necessary to decode GOP (m+1) leading ahead of GOP (m+2) by a GOP as described earlier. In the process to decode GOP (m+2), the processing to decode picture B14 to be decoded last among the pictures composing GOP (m+2) must be completed by a time t4, at which picture B14 is to be displayed, as shown in the picture-decoding order 213 of FIG. 3. Thus, the process to decode GOP (m+2) must be started before a time leading ahead of the time t4 by a period of time it takes to display pictures composing a GOP. In the picture-decoding order 213 shown in FIG. 3, the time t3 is determined to be the time to start the process to decode GOP (m+2). By the same token, the process to decode GOP (m+1) leading ahead of GOP (m+2) by a GOP must be started before a time leading ahead of the time t3 by a period of time it takes to display pictures composing a GOP. In the picture-decoding order 213 shown in FIG. 3, the time t2 is determined to be the time to start the process to decode GOP (m+1).
The time t2 is a time at which MPEG decoder #1 ends the process to decode GOP (m). Thus, as soon as the process to decode GOP (m) is ended, the process to decode GOP (m+1) can be started immediately.
As described above, if the shortest cut duration is set at a period of time it takes to display a GOP, it is possible to reproduce results of editing an MPEG stream by using two MPEG recorders for carrying out the so-called alternate decoding process of GOPs between an edit point and the next edit point.
In accordance with the method disclosed in Japanese Patent Laid-open No. 2002-101379, however, the shortest cut duration, which is a minimum duration required between a start edit point and an end edit point, must be the period of time it takes to display a GOP, raising a problem of restricted freedom of editing.